Rue Gilt Groupe is a fashion eCommerce company located in Boston, MA, that has 50M members and daily flash sales on millions of products. Our Data Science team is a tight-knit group of Data Scientists and Machine Learning Engineers who work full-stack on cloud-native architectures to deliver DS and ML services, heavily utilizing Apache Spark and AWS. This post focuses on some recent updates we incorporated into one of our stacks built for big data applications to add support for running the latest and greatest deep learning based algorithms and models. This architecture provides us with the flexibility to pick the right framework at any step of Machine Learning and unlock scalable deep learning pipelines with minimal MLOps code. At the same time, it also provides the flexibility to transition to any MLOps platform without a lot of future ML code changes.

As you may have experienced in the past, or probably will at some point, running out of memory is a very common issue in Data Science. Due to the large facets of businesses, it is not uncommon to create datasets with over 10,000 or more features. We may choose to process such dataset with tree-algorithms. Deeplearning, however, easily engineers features more automatically and process them into a model of your choice. An often occurring problem is figuring how to train your favorite model within a respectful amount while processing such a huge amount of data.

Apache Spark and TensorFlow are both open-source projects that have made significant impact in the world of enterprise software in recent years. TensorFlow provides a foundational framework for running distributed numerical computations, such as deep learning algorithms, while Spark is a general Hadoop-like, large-scale data processing framework that's also a popular choice for more traditional machine learning algorithms using MLlib. Google Cloud Platform offers managed services for both Apache Spark, called Cloud Dataproc, and TensorFlow, called Cloud ML Engine. Both of these services deliver the power of their respective open-source frameworks in a managed environment, letting you focus on the data science while we worry about the operations. Intuitively, there is some overlap -- Spark provides a framework for big data computations, and the type of datasets that power TensorFlow algorithms tends to be large.